United States P&tenalt [19] [U] Patent Number: 4,974,256 

Cyr et aL [45] Date of Patent: Nov. 27, 1990 



[54] LOAD BALANCING AND OVERLOAD 
CONTROL IN A DISTRIBUTED 
PROCESSING TELECOMMUNICATIONS 
SYSTEM 

[75] Inventors: Bernard L. Cyr, Aurora, 111.; Joseph 
S. Kflufmnn, Holmdel; Tzongyu P. 
Lee, Edison, both of N.J. 

[73] Assignee: AT&T Bell Laboratories, Murray 
Hill, N.J. 



[21] 


Appl.No.: 375,102 




[22] 


Filed: Jun. 30, 


1989 


[51] 


Int. C1.5 


.. H04M 11/00; H04Q 3/66; 






H04Q7/04 


[52] 


U.S. CI 


379/113; 379/59; 






379/269 


[58] 


Field of Search 


379/113, 269, 59 


[561 


References Cited 



U.S. PATENT DOCUMENTS 



4,611,322 9/1986 Larson et al 370/60 

4,658,098 4/1987 Wegmann 379/113 

4,670,899 6/1987 Brody et al 379/60 

4,737,983 4/1988 Frauenthal et al 379/221 



4,788,718 11/1988 McNabb et al 379/113 

4,790,005 12/1988 Hauselka et al 379/244 

4,827,499 5/1989 Warty et al 379/58 

Primary Examiner— Thomas W. Brown 
Attorney, Agent, or Firm — Scott W. McLellan 

[57] ABSTRACT 

This invention relates to a method of allocating new 

* telecommunications cells to one of a plurality of proces- 
sors, and to control acceptance of new calls when in 
overload. The method is based on measuring the real 

* time occupancy of each processor periodically, and 
^allocating new caHs for the following period in such a 

way as to attempt to make each processor's occupancy 
approach the average occupancy of all the processors. 
An overload state is detected when the average occu- 
pancy of the processors in one period exceeds a prede- 
termined threshold, e.g., 90%. When this happens, load 
is shed in an amount calculated to restore die average 
occupancy to the threshold value over a number of 
periods of time. Advantageously, this method optimizes 
the call processing capacity of the processors. 

15 Claims, 2 Drawing Sheets 
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LOAD BALANCING AND OVERLOAD CONTROL 
IN A DISTRIBUTED PROCESSING 
TELECOMMUNICATIONS SYSTEM 

TECHNICAL FIELD 

This invention relates to work allocation to proces- 
sors in a distributed processing system. 

PROBLEM 

In certain stored program controlled telecommunica- 
tions switching systems, such as AT&Ts AutoplexTM 
1000 system for serving cellular mobile radio telecom- 
munications traffic, a plurality of processors are avail- 
able for performing the data processing operations nec- 
essary to control each call in the system. Each of the 
plurality of processors is able to perform identical call 
processing functions required to serve such calls so that 
any new call can be served by any of these processors. 20 

In such an arrangement, in order to maximize system 
capacity and minimize call setup delays, it is important 
to have a quality method of allocating each new call to 
an appropriate processor. It is also important to have a 
quality method for responding to overload conditions, 
so that the telecommunications traffic handled when 
subjected to overload is equal to or at least close to the 
maximum amount of traffic that the system can handle 
when not in overload and that additional calls in over- 
load do not interfere with the processing of existing 
calls or those new calls already accepted by the system. 

The amount of processor time required for each call 
varies substantially depending in the case of cellular 



capacity of a plurality of processors to maximize the 
total system capacity. 

SOLUTION 

5 The above problem is solved and an advance is made 
over the prior art in accordance with this invention 
wherein, in an ^illustrative embcdiment^e^average.real^ 
^time work OTOUpancy of each proc essor is measured J 

jnexj|rjeri^ 

/ equalize the occupancy of all the processors durmgAhaTIL J 
4peJ|>d^In this illustrative embodiment, each of the pro- 
jectors' measure their occupancy simultaneously and in 
15 synchronism and are polled periodically by a call allo- 
cation processor. The call allocation processor adjustsp 
s the fraction of new calls to be allocated to each procesf 
sor during the next perioc|by reducing that fraction for 
processors whose occupanjjr exceeds the average, and 
increasing the fraction for processors whose occupancy 
is less than the average. The fraction Si/ of calls to be 
allocated during period 1 to the i'th processor is equal to 
Si/=So/-l-(A— a/)Gi, wherein So/ equals the fraction of 
new calls allocated to that processor during period 0 
(the previous period), A is the average occupancy of all 
the processors during the previouspenod, a/ is the occu- 
pancy of the i'th processor during the/previous period, 
and Q\ is a feedback gain factor, determined by simula- 
tion and/or experience. Following implementation of 
the algorithm, our invention provides for a post solution 
resuh^of a new call allocation split among all the call 
control processors. Advantageously, such an arrange- 
ment makes it possible to divert traffic from heavily 



25 



30 



radio on the number of cell boundaries and switching ,T£h ^ T , I 

system boundaries that a mobile traverses in the course^ 35 Of^_»f overloaded) processors toward 

. ^ - fc . " .T ~- 1, ZT more hghtiy loaded processors and tends to spread the 

of its call, and on the features invoked by the call These load ^ ^ ^ pTOying ^ it 

t^°Z C Zl* ESS Whea J^ "*« * ^sivenesso^^^ 

1* 7 1 1 ' ' m * maximized because a balanced l^riLbes 
allocated to a particular processor can be controlled ^ — 

accurately, the amount of processing time required over 



40 



a period of minutes for processing these calls will still 
vary by a substantial quantity. 

In a prior art arrangement for the Autoplex 1000 
system, described, for example, in Warty et al: U.S. Pat. 
No. 4,827,499, the fraction of new load which was allo- 
cated to each processor was determined initially by 
initialization parameters that were subject to modifica- 
tion if any processor went into an overload condition. 
Overall system overload (as opposed to individual pro- 
cessor overload) was recognized when the fraction of 
individual processors that were overloaded exceeded a 
threshold; then the percentage of new calls that were 
rejected was gradually and linearly increased at a fixed 



belSojaffibut^ 
(eg-, call as^^ 
variable anil unpre^ctable^This scheme automatically 
accounts"for~such siae loads by shifting the call load, 
45 which is the majority of the load at any time in any of 
the processors, on the aggregate of processors. 
Jfjhej^yerage occupancy of theSr&essors exceeds a 3 
^hrielhplS ? (i.^ 
Z.overld1Bifs1^^ 

50 the fraction of new calls accepted isle^ than^ l), then a 
determination is made to allow a fraction Pi of-offered: _ 
caUs to'be^ceptedrwhere Pi is the percentage previ- 
ously allowed (Pq), plus a multiple of the difference 
between T, the level of average occupancy of the pro- 



rate as long as the system was in overload and decreased 55 cessors at the threshold of overload, and A, the average 

in tliA earns niati uik«. *k. ./ t 1 . _ _ ° 



in the same way when the system went out of overload, 
Using this arrangement, individual processors close to 
overload tended to be allocated too much traffic while 
underloaded processors were allocated too little, and 
the response to system overload was less than optimum. 60 

In view of the foregoing, a problem of the prior art is 
that in a distributed processing telecommunications 
control system wherein calls may be allocated to differ- 
ent processors, there is not a quality arrangement for 
optimizing the traffic capacity of each processor and 65 
the performance of the system as a whole in overload. 
More generally, in distributed processing systems, there 
is not a good system for equitably distributing the traffic 



measured occupancy. In one specific embodiment, 



1 + (T - A) 



A 



where G2 is a feedback stabilization factor. Simulations 
on the Autoplex 1000 system have indicated that a value 
of G2=3 is appropriate. For example, if T=0.9, 
A =0.95 and G2=3, and Po=l, signifying a sudden 
overload at the 0.95 occupancy load, where 0.9 repre- 
sents the threshold, then Pj =0.85, or 15 per cent of load 
is shed. This fraction is then broadcast from the allocat- 
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ing processor to all new call request generating systems of working call control processors in the system, is 
to request that they accept the specified fraction of new appropriate. The value of Gi must be changed if one or 
calls and reject or shed the remainder. Advantageously, more of the CCPs is temporarily out of service. The 
such an arrangement allows the system to go into a load period over which occupancy measurements are taken, 
shedding mode quickly when necessary, and to stop 5 and the call splits are adjusted, is also important; simula- 
shedding load unnecessarily as soon as possible. tion studies for the Autoplex 1000 indicate that 10 see- 
Accordingly, this invention is an arrangement for onds is a good choice for that system. The new alloca- 
distributing new load to one of a plurality of processors tion increases the fraction of new calls assigned to pro- 
of a distributed processing system by measuring the cessors whose occupancy was below the average and 
occupancy of each of the processors and adjusting the 10 decreases the fraction for processors whose occupancy 
fraction of new load to be allocated to each processor to was above the average. Based on these new calculated 
attempt to equalize the occupancy of each of the pro- splits, a call allocation sequencer is regenerated to re- 
cessors. fleet the new splits; the call allocation sequencer is a 

BRIEF DESCRIPTION OF THE DRAWING , € tobl . e fo * ^mentation of variable percentage round- 

15 robin allocation. 

FIG. 1 is a block diagram of the control system of a Methods for implementing such variable round-robin 

prior art cellular telecommunications system; allocations are well known in the art. In applicants* 

FIG. 2 is a flow diagram of a program executed in a specific implementation, given the splitting percent- 
call allocating processor for determining the allocation ages> a deterministic CCP call allocation sequence is 
of calls to processors in a next interval, and for deter- 20 generated that approximates the splitting percentages, 
mining how much load to shed in overload; and For example, if a CCP's splitting percentage is 25, one- 

FIG. 3 is a flow diagram of an occupancy measure- quarter of the entries in the assignment sequence will be 

ment and reporting process executed in all call control g i ven to ^ CCP. All sequence entries of a given CCP 

processors. are eve „iy distributed throughout the sequence (to the 

DETAILED DESCRIPTION 25 degree possible). This ensures that calls are distributed 

as "evenly" as possible, thus minimizing mean queue 

In the Autoplex 1000 system as described, for exam- lengths ^ delays ^ ^ ccp clearIy tne 

pie, m U.S. Pat No. 4,827,499, a plurality of call control length of ^ sequence determines the accuracy with 
processors (CCPs) is used for serving calls. FIG. 1 is a which ^ splittin percentages can be approximated. In 
Mock diagram of the control system of the Autoplex 30 this embodiment, a length of 64 was selected. This was 
1000 system, modified according to the present inven- for a case of a ja^^ of 6 processors. More gener- 
tion. Any new c^ conung into the system can be as- ^ for n processorSi a lengtn of 10 „ a p pears to be 
signed to any of these CCPs (5,7). The assignment is appropriate . The assignment algorithm walks 
performed by one of these call control processors, tnrou ^ this ^ ^g^g the next ^ to ^ next 
called for convenience, an allocatton/call control pro- 35 ccp m tne nce (wrapping aiaaai when neces . 
cessor 5 (A/CCP). As shown m FIG. 3, each CCP 5,7 s j 

measures its own average occupancy over a period, in ^ ^ orithm ^ generates me cal , ^g^t 
dp ; case, ten seconds. HOI also shows a plurality of ^ tt for ^ ce element . % ^ 

digital circuit switches (DCS) 9 for switching callcon- ccp(j) ^ mioimizes the ^ eteace between the pro- 
nections. TheDCS generate new call requests. These 40 ^ of entries aUocated to CCPffl; in the 

a *L?«^ 0 ^p^ .»-""NkUn»g first j - 1 sequence entries and the call splitting percent- 

xmessage:swit_ch;mfe 3 for-commu-^ ^ CCP(i) 

nicating among prc<^essors^mdjwitches. The new call 

^requests are transmitted to A/CCP 5 7 for allocation to 

^/itself OT One of th e Other CCPs 7 ; _The~mvention-is,imv45 for each sequence element j choose i to minimize 

plemented by progranT6, part of which is described in (d(i.j-i/j>Si/ where: 

FIG. 2 and which resides in A/CCP 5; and program 8, d(i, j-l)/j is the proportion of sequence entries already given 

included within program 6, described in FIG. 3, which toCCP(i) in the first j-i sequence entries 

resides in each of the processors 5,7. S| ' is the call split for CCP(i) 

Periodically, herein every ten se conds , the A/CCP 50 
polls each of the o ther CCPs a ndTtselfto receivelhe At each call load balancing interval, the splitting proba- 
measurement of occupancy (action block 101, FIG. 1) bilities are re-calculated and the call assignment se- 
andjqstart another periodof occupancy measurement. quence for the next interval is generated. 

/The A/CCP^alculates'the average occupancy (A) by Blocks 109, 111, and„113_relate to overload~control. x 
averaging the individual r occupancies 'a/ (action block' 55 Test 109 checks whether the* system is already in the^ 

^103). For loaa balancing purppsesrthe A/CCP-then^ i calUhedding m ode or if the average occupancy exceeds . 

^calculates' a hew split of allocation of new calls jor the a threshold T. The latter test is the test used for entering- 
Scorning period according to the algorithm Sj^So/+- „ the call shedding mode wherein a fraction of new calls > 
(A— a/)GY, wherein Si/,- So/ represent the fraction of " is simply not accepted by the system. In this particular 7 
calls allocated to the i'th processor during the next 60 embodiment, calls are shed at random and no prefer- 
interval and the previous interval, respectively, A is the ence is made for either incoming land-originated mobile 
average occupancy of the processors as calculated in terminating calls or mobile originated calls. It might 
action block 103, a/ is the occupancy reported by the i'th under some circumstances be desirable to shed more of 
processor, and Gi is a feedback gain parameter which is the mobile originated calls when call shedding is neces- 
tuned to yield the desired dynamic response. Gi is a 65 sary, since no resources have yet been used for such 
function ofthe number of call control processors. Simu- calls whereas land-originated calls have already had 
lation studies for the Autoplex 1000 system have indi- substantial resources used in setting up the connection 
cated that a value for Gi = 1/n, where n is the number to a mobile switching office. However, the disadvan- 
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tage is that if this were done, then under overload mo- 
bile units would be able to receive most calls but would 
have a much lesser chance of being able to originate 
calls and this may be considered undesirable service to 
the mobile units. 

If the result of test 109 is negative, then there is no 
need to perform any call shedding and the actions of 
this particular sequence is terminated (action block 
115). If the result of test 105 is positive, then a new 
calculation is made for the percent of calls to be shed 
based on the present value of A. This calculation is 
shown in action block 111, is based on the algorithm 



10 



-4 



i + <r - A) 



A 



15 



wherein Pi, Po represent the new and previous percent- 
age of new calls accepted (if the system is just entering 
overload, then Po= 1), T is a predetermined threshold 20 
of average occupancy for invoking overload control, 
and G2 is a gain factor which is tuned to yield the de- 
sired dynamic response. In the exemplary embodiment 
of the invention based on simulations, the value selected 
for G2 is 3, and the value selected for T is 0.9. In this 25 
calculation, Pi is truncated so that it can never exceed 1 
and is never less than 0.05. The fraction Di of calls shed, 
1-Pi, is: 



Di = 1 



-4 



\ + (T~A) 



■]• 



30 



A value of G2 above 1 is desirable in order to respond 
promptly to sudden overloads. The value of A is limited 35 
to 1 and if A is at that value, the actual overload may be 
far greater than that implied by the difference between 
1 and T. Therefore, a sharp decrease in accepted load, 
made possible by a high value of G2, protects the system 
against excessively long message queues and processing 40 
delays, which result in degraded performance in han- 
dling both old and new calls. A high value of G2 also 
leads to a more rapid recovery from overload controls. 
It is possible that field experience will suggest the use of 
a high value of G2 for imposing overload controls, i.e., 45 
decreasing Pi whereas a lower value of G2 may be 
appropriate for releasing overload controls, i.e., increas- 
ing Pi. 

The value of Pi is then broadcast to all the nodes 
which transmit call messa g es to the processors (action 50 
block, 113). These nodes will then delete a fraction 
1— P, of new call messages and simply not forward 
-these deleted messages to the A/CCP for allocation 'to 
processors, f 

The deletion process can be carried out by the fol- 55 
lowing algorithm which is executed for each new call 



1, the call message is dropped and A is decremented by 
1; otherwise the call message is sent 

FIG. 3 shows the measurement process which is 
executed in each CCP7 and in A/CCP3. The length of 
real time segments is accumulated throughout the per- 
iod of ten seconds (action block 201). The processor 
then receives a polling request from A/CCP5 (action 
block 203). In response, the processor converts the 
accumulated number into a percentage, representing 
the processor occupancy, transmits this occupancy 
measure to the A/CCP (action block 205) and resets the 
value of the real time segment accumulator to begin 
another cycle of measurement 

It is to be understood that the above description is 
only of one preferred embodiment of the invention. 
Numerous other arrangements may be devised by one 
skilled in the art without departing from the spirit and 
scope of the invention. The invention is thus limited 
only as defined in the accompanying claims. 
We claim: 

1. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

receiving a measurement of real time work occu- 
pancy of each of said plurality of processors; and 
responsive to said received measurements, adjusting 
an allocation of new calls to each processor to 
equalize the occupancies of each of said processors. 

2. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

receiving a measurement of real time work occu- 
pancy of each of said plurality of processors; 
responsive to said received measurements, adjusting 
an allocation of new calls to each processor to 
equalize the occupancies of each of said processors; 
detecting that an average of said measured occupan- 
cies of said processors exceeds a predetermined 
threshold; and 
shedding a fraction of new calls, said fraction deter- 
mined to change an average occupancy of said 
processors to approximately equal said threshold. 

3. The method of claim 2 wherein said shedding com- 
prises adjusting said shed fraction by a multiple of a 
difference between a predetermined threshold value 
and said average of said measured occupancies of said 

y plurality of processors. 

4. The method of claim 2 wherein said shed fraction 
is determined periodically by an algorithm: 



A = A + (1 - P), 60 
if A > I 

then drop the call message; 

A = A - 1 

else 

send the call message 

65 



Briefly, a parameter A, initially 0, is incremented by 
the fraction of calls to be deleted. Whenever A exceeds 



wherein Di represents said shed fraction for a next 
period, Po represents a fraction of calls accepted during 
a period immediately prior to said next period, T repre- 
sents a predefined constant representing an average 
occupancy of said plurality of processors at a threshold 
of overload of said telecommunications control system, 
A represents average measured occupancy of said plu- 
rality of processors during said prior period and G2 
represents a predetermined gain factor. 
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5. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

receiving a measurement of real time work occu- 
pancy of each of said plurality of processors; and 

responsive to said received measurements, adjusting 
an allocation of new calls to each processor to 
equalize the occupancies of each of said processors; 

wherein said adjusting is performed periodically and 
comprises adding to an allocated fraction for each 
processor for a prior period a predetermined multi- 
ple of a difference between an occupancy of said 
each processor during said prior period and an 
average occupancy of said plurality of processors 
during said prior period. 

6. The method of claim 5 wherein said allocated frac- 
tion for an i'th one of said plurality of processors is 
detennined by an algorithm: 



10 



15 



20 



wherein Si/, So/ represent an allocated fraction for said 
i'th processor for a next and said prior period, respec- 
tively, A represents an average occupancy of said plu- 
rality of processors during said prior period, a/ repre- 
sents an occupancy of said i'th processor during said 
prior period, and Gi represents a predetermined gain 
factor. 

7. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

receiving a measurement of real time work occu- 
pancy of each of said plurality of processors; and 

responsive to said received measurements, adjusting 
an allocation of new calls to each processor to 
equalize the occupancies of each of said processors; 

wherein said occupancy of each of said processors is 
measured periodically over the same period, and 
said adjusting is performed at a beginning of each 
period. 

8. The method of claim 7 wherein each of said proces- 
sors is polled periodically to transmit said measurement. 

9. A method of assigning work to a plurality of pro- 
cessors comprising: 

detennining a work occupancy for each of said pro- 
cessors; 

responsive to said occupancy determining, adjusting 
allocation fractions of new work for each of said 
processors toward work occupancy equalization 
among said processors; and 

thereafter allocating new work to said processors 
based on said adjusted allocation fractions. 

10. In a distributed multiprocessing system compris- 
ing a plurality of processors each for performing data 
processing operations in response to requests, a method 60 
of allocating new requests to ones of said plurality of 
processors comprising the steps of: 

measuring a work occupancy of each of said plurality 

of processors; 
responsive to said measuring, periodically adjusting 

allocation fractions of new work requests for each 

processor toward work occupancy equalization 

among said processors; and 



25 



30 
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thereafter, allocating new work to said processors 
based on said adjusted allocation fractions. 

11. The method of claim 10 wherein said adjusting 
comprises adding to an allocation fraction for each 
processor for a prior period a predetermined multiple of 
a difference between an occupancy of said each proces- 
sor during said prior period and an average occupancy 
of said plurality of processors during said prior period. 

12. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

measuring a work occupancy of each of said plurality 
of processors; 

periodically adjusting a fraction of new calls to be 
allocated to each processor in such manner that the 
occupancies of each of said plurality of processors 
are adjusted toward equalization; 

detecting that an average of said measured occupan- 
cies of said processors exceeds a predetermined 
threshold; and 

shedding a fraction of new calls, said fraction deter- 
mined to attempt to achieve an average occupancy 
of said processors approximately equal to said 
threshold; 

wherein said shedding comprises adjusting said shed 
fraction by a multiple of a difference between a 
predetermined threshold value and a measured 
average occupancy of said plurality of processors; 

wherein said shed fraction is determined periodically 
by an algorithm: 

wherein Di represents said shed fraction for a next 
period, Po represents a fraction of calls accepted 
during a period immediately prior to said next 
period, T represents a predefined constant repre- 
senting an average occupancy of said plurality of 
processors at a threshold of overload of said tele- 
communications control system, A represents aver- 
age measured occupancy of said plurality of pro- 
cessors during said prior period and Gi represents 
a predetermined gain factor; 

wherein said adjusting comprises adding to an allo- 
cated fraction for each processor for a prior period 
a predetermined multiple of a difference between 
an occupancy of said each processor during said 
prior period and an average occupancy of said 
plurality of processors during said prior period; 

wherein said allocated fraction for an i'th one of said 
plurality of processors is calculated by a formula: 

wherein St/, So/ represent an allocated fraction for 
said i'th processor for a next and said prior period, 
respectively, A represents an average occupancy 
of said plurality of processors during said prior 
period, a/ represents an occupancy of said i'th pro- 
cessor during said prior period, and Gi represents a 
predetermined gain factor; and 
wherein said occupancy of each of said processors is 
measured periodically over the same period, and 
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said adjusting is performed at a beginning of each 
period. 

13. In a distributed multiprocessing telecommunica- 
tions control system, means for allocating new calls to 
ones of a plurality of call control processors, compris- 
ing: 

a plurality of call control processors, each operative 
under the control of a first program, for measuring 
its own real time work occupancy; and 

processor means, operative under the control of a 
second program and responsive to said measuring 
real time work occupancy of each of said call con- 
trol processors, for adjusting an allocation of new 
calls to each of said call control processors to 



equalize said real time work occupancies of each of 
said call control processors. 

14. The means for allocating new calls of claim 13 
wherein said call control processors are further opera- 

5 tive under the control of said first program for measur- 
ing said occupancies periodically and for reporting said 
measurements to said processor means periodically. 

15. The means for allocating new calls of claim 13 
wherein said processor means is further operative under 

10 the control of said second program to determine 
whether an average of said measured occupancies ex- 
ceeds a predetermined threshold, and, responsive to 
said determination, for controlling the shedding of a 
fraction of new calls, said fraction determined to change 

15 an average occupancy of said call control processors to 

approximately equal said threshold. 
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